Abstract
Line drawing colorization is an indispensable stage in the image painting process, however, traditional manual coloring requires a lot of time and energy from professional artists. With the development of deep learning techniques, attempts have been made to colorize line drawings by means of user prompts, text, etc, but these methods also seem to require some manual involvement. In this paper, we propose a reference-based colorization method for cartoon line drawings, which uses a more stable diffusion model to automatically colorize line drawings to improve the quality of the generated images. In addition, to further learn the color of the reference image and improve the quality of the colorized image, we also design a two-stage training strategy. To ensure the generality of the model, in addition to the 17,769 benchmark datasets shared on the Kaggle, we used the cartoon dataset provided by the competition in the fine-tuning stage and created a small garment dataset. Finally, we illustrate the effectiveness of the model in reference-based automatic coloring through a large number of qualitative and quantitative experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, L., Wan, L.: CTUNet: automatic pancreas segmentation using a channel-wise transformer and 3d u-net. Visual Comput. 39, 1–15 (2022)
Ci, Y., Ma, X., Wang, Z., Li, H., Luo, Z.: User-guided deep anime line art colorization with conditional adversarial networks. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1536–1544 (2018)
Dhariwal, P., Nichol, A.: Diffusion models beat GANs on image synthesis. Adv. Neural. Inf. Process. Syst. 34, 8780–8794 (2021)
Dowson, D., Landau, B.: The fréchet distance between multivariate normal distributions. J. Multivar. Anal. 12(3), 450–455 (1982)
He, M., Chen, D., Liao, J., Sander, P.V., Yuan, L.: Deep exemplar-based colorization. ACM Trans. Graph. (TOG) 37(4), 1–16 (2018)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)
Hu, X., Zhang, J., Huang, J., Liang, J., Yu, F., Peng, T.: Virtual try-on based on attention u-net. Vis. Comput. 38(9–10), 3365–3376 (2022)
Kim, H., Jhoo, H.Y., Park, E., Yoo, S.: Tag2pix: line art colorization using text tag with secat and changing loss. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9056–9065 (2019)
Kim, T.: Anime sketch colorization paired dataset from danbooru (2018). https://www.kaggle.com/datasets/ktaebum/anime-sketch-colorization-pair
Lee, J., Kim, E., Lee, Y., Kim, D., Chang, J., Choo, J.: Reference-based sketch image colorization using augmented-self reference and dense semantic correspondence. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5801–5810 (2020)
Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: ACM SIGGRAPH 2004 Papers, pp. 689–694 (2004)
Liu, X., Wu, W., Li, C., Li, Y., Wu, H.: Reference-guided structure-aware deep sketch colorization for cartoons. Comput. Visual Media 8, 135–148 (2022)
Liu, Y., Qin, Z., Wan, T., Luo, Z.: Auto-painter: cartoon image generation from sketch by using conditional wasserstein generative adversarial networks. Neurocomputing 311, 78–87 (2018)
Nazir, A., et al.: Ecsu-net: an embedded clustering sliced u-net coupled with fusing strategy for efficient intervertebral disc segmentation and classification. IEEE Trans. Image Process. 31, 880–893 (2021)
Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2337–2346 (2019)
Qu, Y., Wong, T.T., Heng, P.A.: Manga colorization. ACM Trans. Graph. (ToG) 25(3), 1214–1220 (2006)
Saharia, C., et al.: Palette: image-to-image diffusion models. In: ACM SIGGRAPH 2022 Conference Proceedings, pp. 1–10 (2022)
Silva, F.C., de Castro, P.A.L., Júnior, H.R., Marujo, E.C.: Mangan: assisting colorization of manga characters concept art using conditional GAN. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 3257–3261. IEEE (2019)
Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Wen, Y., et al.: Structure-aware motion deblurring using multi-adversarial optimized cyclegan. IEEE Trans. Image Process. 30, 6142–6155 (2021)
Zhang, L., Ji, Y., Lin, X., Liu, C.: Style transfer for anime sketches with enhanced residual u-net and auxiliary classifier GAN. In: 2017 4th IAPR Asian Conference on Pattern Recognition (ACPR), pp. 506–511. IEEE (2017)
Zhang, Q., Wang, B., Wen, W., Li, H., Liu, J.: Line art correlation matching feature transfer network for automatic animation colorization. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3872–3881 (2021)
Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40
Zhang, R., et al.: Real-time user-guided image colorization with learned deep priors. arXiv preprint arXiv:1705.02999 (2017)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
He, J. et al. (2024). Reference-Based Line Drawing Colorization Through Diffusion Model. In: Sheng, B., Bi, L., Kim, J., Magnenat-Thalmann, N., Thalmann, D. (eds) Advances in Computer Graphics. CGI 2023. Lecture Notes in Computer Science, vol 14496. Springer, Cham. https://doi.org/10.1007/978-3-031-50072-5_29
Download citation
DOI: https://doi.org/10.1007/978-3-031-50072-5_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-50071-8
Online ISBN: 978-3-031-50072-5
eBook Packages: Computer ScienceComputer Science (R0)